Stream Aggregation with Compressed Sliding-Windows

High performance stream aggregation is critical for many emerging applications that analyze massive volumes of data. Incoming data needs to be stored in a sliding window during processing, case the functions cannot computed incrementally. Updating with new incoming values and reading it feed are two primary steps aggregation. Although updates can supported efficiently using multi-level queues, frequent aggregations remain bottleneck as they put tremendous pressure on memory bandwidth capacity. This article addresses this problem by enhancing StreamZip, dataflow engine able compress windows. StreamZip deals number control dependency challenges integrate compressor pipeline alleviate posed aggregations. In addition, incorporates caching mechanism dealing skewed-key distributions stream. doing so, offers higher throughput well larger effective capacity support problems. supports diverse compression algorithms offering both lossless lossy integers floating-point numbers. Compared designs without compression, achieve up 7.5× 22× throughput, while improving 5× 23×, respectively.